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Vortex lattice structures in Rashba noncentrosymmetric superconductors in magnetic fields par- 
allel to the basal plane (H _L c) are examined based on the BCS-like Hamiltonian and the resulting 
Ginzburg-Landau functional. Due to the momentum dependent anisotropy of the Zeeman effect 
' induced by the broken inversion symmetry, the vortex lattice in higher fields generally shows some 

unidirectional modulation of Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) type orienting in the plane 
f«) ■ perpendicular to H. However, the direction of the modulation and the lattice structure depend 

' significantly on the underlying pairing symmetry: When the mixing between spin singlet and triplet 

pairing components is negligible, the resulting modulated structure tends to have reflection sym- 
metry, while the vortex lattice in systems with a significant singlet-triplet mixing has no reflection 
symmetry in most cases. The latter result implying the presence in real materials of two degenerate 
orientations of the lattice structure separated by domain walls may be relevant to the extremely low 
magnetic decay rate observed in CePtsSi. 
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PACS numbers: 74.20.Fg, 74.20.Rp, 74.25.Qt, 74.70.Tx 
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O ; I. INTRODUCTION 

ill ' 

0H, The Pauli paramagnetism has crucial effects on superconducting vortex states. Reflecting the pictures [j], 0, 0] 
expected in the vortex free (Pauli) limit, it tends to change the character of the mean field superconducting transition 
occurring on the depairing field H^iT) from the conventional second order into a first order one [J] and induces an 
additional spatial modulation 0, [f| as a reflection of a Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) state in the Pauli 
limit. Further, such a spatial variation of the pair-field, i.e^ superconducting order parameter, due to the vortices and 
the FFLO modulation can also induce a parity mixing @, 0] in the vortex state. In the ordinary superconductors with 
inversion symmetry, however, these effects are quite weak, and, in particular, it is not expected that the field-induced 
parity mixing [6|,|7( changes the vortex phase diagram qualitatively. Besides this, effects of the paramagnetic depairing 
on the vortex lattice structure have not been discussed until recently. It has been found [|| that, in systems where the 
l— "™ '■ paramagnetic effect is strong enough, it weakens an anisotropy in the vortex lattice structure reflecting the pairing 
^vq , symmetry or the band structure and stabilizes the isotropic triangular lattice structure. 

■ In this paper, we study possible relations between the pairing symmetry and the vortex lattice structure in noncen- 
C***~ \ tro symmetric superconductors with spin-orbit coupling of Rashba type in magnetic fields parallel to the basal plane 
(H 1c). It is found that, in contrast to the above-mentioned consequences in centrosymmetric superconductors with 
inversion symmetry, spatial modulations induced by the paramagnetic depairing and a field-induced parity mixing oc- 
cur even for relatively lower values of the Maki parameter § v^-Horb(0)/-Hp(0), and that the vortex lattice structure 
in Rashba superconductors strongly depends upon the details of the orbital component of the Cooper pairing state, 
where iJ or b(0) and -Hp(O) are the orbital and Pauli limiting fields at zero temperature, respectively. The main origin of 
these intriguing effects is an anisotropic Zeeman energy for quasiparticles stemming from the antisymmetric spin-orbit 
I* ■ coupling (ASOC) peculiar to noncentrosymmetric superconductors. This ASOC is expressed as an additional term 



X' «soc= 2^ (% a Cgk- OTajCkj (1) 

c3 | 

in the electronic Hamiltonian, where £ is the energy scale measuring the magnitude of ASOC, and Ck,a is the annihi- 
lation operator of conduction electron with momentum k and spin projection a. The type of ASOC is defined by the 
vector gk corresponding to the Fourier transform of the ASOC, and, in Rashba superconductors, it can be expressed 
as (kx z)/k F , where kp is the Fermi wavenumber, and the c-axis corresponds to the z-direction. Due to this term, the 
quasiparticle band of our interest in the normal state is split into two pieces The strength of ASOC is measured 
by the dimensionless quantity 

SN^^l^K, (2) 
Ni + N 2 E F 

expressing a normalized difference between the density of states on the split two bands, N\ and N 2 . Hereafter, each 
Fermi surface (FS) or band will be specified by the indices a — 1 and 2. Through the present paper, the ideal limit 
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FIG. 1: (Color online) Fermi surfaces (FSs) in a magnetic field of (a) a centrosymmetric superconductor and (b) a Rashba 
noncentrosymmetric one. The colored dots indicate partners of Coooer pairings on the FSs. In the case (b) satisfying eq.([3|, 
each Cooper pair is formed on the same FS, and the paramagnetic depairing is effective only through the relative displacement 
of the two FSs of which the magnitude is 2Qo = 2)xH/vf (see sec. I). In the limit in which one of the two FSs is irrelevant to 
superconductivity, such an FS-displacement induced by the Zeeman effect is trivial and reduces to the orbital-limited case. 



SN — > corresponding to the limit of large band width {Ep — > oo) is often considered in order to understand physical 
origins of structural changes of the vortex lattice. As a result of the coupling in the spin space between the ASOC 
and the original Zeeman terms, the quasiparticle Zeeman energy on the split FSs becomes k-dependent. Further, we 



focus in this article on noncentrosymmetric superconductors satisfying the condition 11 | 



max(T,^ff)<C<-E^, (3) 

where [iH is the Zeeman energy in centrosymmetric case. Under this condition, the Zeeman energy in noncentrosym- 
metric materials becomes highly anisotropic in the momentum space so that the paramagnetic depairing is ineffective 
in H || c fill] , where other high order corrections in /i-ff/C were neglected. This ASOC-induced anisotropy in the 
Zeeman energy makes emergence of modulated vortex states in Rashba superconductors easier compared with that 
in centrosymmetric systems. 

Before proceeding further, we introduce two key parameters for determining the type of modulations in vortex 
states in Rashba superconductors. One is SN defined in cq.([2]). However, we note that, as indicated in Fig. 1(b), a 
modulation induced by the paramagnetic depairing is expected even in \SN\ — ► limit. The other is a parameter 
measuring closeness between the most attractive singlet and triplet pairing states and will be expressed in the form 

. 2(N 1 + N 2 ) 

where w ss (wtt) is the strength of attractive interaction in the most attractive pairing state in singlet (triplet) channels. 
In the limiting case with just a single pairing state, \Sw\ vanishes, while it is divergent when both of the two pairing 
channels equally contribute to superconductivity. In the latter case, only one of two FSs contributes to the pairing. 
Then, the pure orbital-limiting case is reached, because a field-induced displacement of the only FS participating in 
superconductivity does not affect the pairing itself (see Fig. 1(b)). 

To clarify why a state modulating in the plane perpendicular to H tends to occur more easily compared with in 
the centrosymmetric case, the h-t phase diagrams (a) in the centrosymmetric case and (b) in Rashba case in SN — > 
limit are compared with each other in Fig. 2, where t — T/T c , and the normalized field h will be defined in the ensuing 
sections. For simplicity, the possibility 4] of the first order H^-transition and of a modulation parallel to H in 
Fig. 2(a) is neglected here. Between the figures (a) and (b), the only difference in the used Hamiltonian is the Zeeman 
energy term: In (a), the ordinary isotropic Zeeman energy /iH • a is used, where a denotes the spin projection, and 
the modulated state, (A) and (B), corresponding to the FFLO state in the Pauli limit appears in higher fields and is 
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FIG. 2: (Color online) Typical h-t phase diagrams of (a) a centrosymmetric superconductor and (b) a Rashba noncentrosym- 
metric one in SN — > limit following from the present theory, where h and t are normalized field and temperature, respectively. 
Each blue curve denotes the _ff C 2(T)-curve, while each red curve is a first order structural transition (FOST) between different 
structures of the vortex lattice. In (a), the possibility of a discontinuous /^-transition and an FFLO modulation parallel to 
the field was neglected to make comparison between (a) and (b) easier. Insets: The images (A), (B), and (C) in each figure are 
real space patterns of the amplitude of the pair-field in the x-z plane at each (h, i) below iZ C 2-curve. The energy gap nearly 
vanishes in the darkest regions. 



separated by a first order transition occurring on the red curve from the conventional triangular lattice (C). On the 
other hand, in (b), the Zeeman energy is influenced by Tt soc and takes the form fiH ■ (k x z). Due to this anisotropy 
in the momentum space of the Zeeman effect, the overall effect of the paramagnetic depairing is weakened in the 
noncentrosymmetric case, as can be seen from the relative enhancement of H C 2 and the narrower FFLO region with 
the structure (A) in (b). However, novel modulated vortex states, the intermediate states (B) and (C), tend to occur 
in (b) due to the momentum dependent Zeeman energy term. That is, contrary to the centrosymmetric case with 
isotropic Zeeman energy, a novel modulated vortex state can be expected to occur in Rashba superconductors even if 
the paramagnetic depairing effect is so weak that the conventional FFLO-likc modulated state (A) does not appear. 
In fact, this is the basic reason why, as will be shown below, various spatially-modulated states can occur dependent 
on the pairing symmetry in Rashba superconductors. 

It should be stressed that the novel spatially-modulated states, (B) and (C) in (b) in intermediated fields, appear 
irrespective of SN and hence, is not relevant to the helical phase modulation [l2l [l3[ argued to occur in the vortex 
free situation in Rashba superconductors. The phase modulation is hidden in the vortex states in Rashba case and 
merely appears as an anisotropy orienting the direction of the novel modulated structure. However, the helical phase 
modulation is observable in noncentrosymmetric superconductors of cubic type and should be detected as a local and 



transverse magnetization 14 1 



In sec. II, the microscopic basis of our calculation is explained. After deriving the Ginzburg-Landau (GL) free energy 
functional, results on phase diagrams in the pure singlet or pure triplet case are shown and discussed separately for 
a couple of pairing states in sec. III. In the next section, the analysis is extended to the case with a singlet and triplet 
mixing, and discussions on relevance of the present results to real systems are given in sec.V. 



II. ELECTRONIC HAMILTONIAN 



We start from the following electronic Hamiltonian 

Hcl = Hsingle + Hint, (5) 

where 

^single = C lL'i ( £kSs i^2 + (C(gf)k + HbH u )((J v ) S1!S2 ) C k , S2 (6) 

k,Sl ,S2 

is the kinetic energy Hamiltonian accompanied by Tt soc and the bare Zeeman term, c k is the creation operator of 
a conduction electron with the wave vector k and the spin index s, jj,H the magnitude of the bare Zeeman energy, 
H = |H|, £k is the bare dispersion of conduction electrons defined with no ASOC, (c At ) Sl;S2 is a Pauli matrix, and 
gk = k x z/kF (see sec. I). At the present stage, the orbital effect of the magnetic field was neglected in eq.([6]). 

Throughout this paper, our calculation is performed within models based on the quasi two-dimensional (Q2D) 
dispersion of quasiparticle energy 

^k= ^-(k 2 x + k 2 y ) + J(l-cos(fc z )), (7) 

where k z = k z d, and d is the period in the z (or c) direction. In most of real tetragonal noncentrosymmetric materials, 
the main Fermi surface (FS) seems to be an ellipsoid. However, effects of discrete layer structure in the original Q2D 
model are not considered hereafter by assuming d -C £, z (0) so that d does not appear in the resulting GL functional, 
and, under this simplification, the GL functional for the Q2D FS is equivalent to that of the ellipsoidal FS, where 
£„(0) (v = x, y, z) is the zero temperature coherence length in the ^-direction. Then, the only parameter characterizing 
FS effects is the anisotropy parameter 7 between the coherence lengths, which is given in terms of J = J/ef by 
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^W)^W) = ^^- (8) 



For this FS, the average of a quantity /(v) over the momentum on the FS is given by 



= «f \J 1 — J(l — cosfc z )cos0k 



fj, = up y 1 — J(l — cosfc z )sin0i c 
w z = J dsmk z , 

where 0k = tan (ky/k x )- Hereafter, we assume fcj? d = ir in order to merely reduce the number of inessential material 
parameters. 

The interaction Hamiltonian takes the following generic form 

Hint = V H M/ «/3:75( k l ) k 2) c k 1 +p/2,a C -k 2 +p/2, / 3 C -k2+p/2,5C k2+p /2, 7 , (10) 
p,ki,k 2 

where the interaction potential W aj 8 .y$(ki,k2) may be expressed as 

i,j=s,t 

with r s (k) = i(7j,Ak, Tt(k) = i(cr B <7^) ■ (gk)^Ak, where Ak expresses a normalized pairing function of the dominant 
component in the spin-singlet channel, and the so-called <i-vector of the spin-triplet component was replaced in r t by 
gk based on the inequality ([3]) [llj. Further, 6 denotes the unit vector parallel to o. Hereafter, we will focus on the 
case in which both of the singlet (s) and triplet (t) channels are attractive, and the matrix lUjj (i, j = s and t) is 
positive definite. 

Due to ASOC, Ti. slng is not diagonalized via the spin states, and the following unitary transformation is needed to 
diagonalize it: 

°ln (£!)-(&)■ (12) 

where / is the field operator of the resulting quasiparticles, and 

u q s ^ = 1 + i(sin0 k o- y - cos^kOa) ^ 

Then, 7i S ingie is represented in terms of quasiparticle states /k, with two split FSs 

^single = X! X! fLa E Kafa,a, (14) 
k a=l,2 

where 

£k, a =£ k + (-l) a+1 |Cgk + MH|. (15) 
The quasiparticle Green's function close to FS a is 

Sa(K ie) * : \ „ . , (16) 

ie - £a + (-l) a /iH • g k 

where £ a is the single particle energy measured from the FS a, and gk = gk/|gk|- Throughout this paper, the field 
configuration H | y is assumed in which the Pauli paramagnetism is effective, and the H || z (H || c) configuration in 
which the Zeeman term vanishes will not be considered. 
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Correspondingly, the interaction Hamiltonian is expressed in the form 



2 

p i,j=s,t 

where the pair-field operators take the form 

*P S) - - E 7 E A * e [1( - 1)a+Vkl /- k+pA « /k+p/ 2 ,a (18) 
k a=l,2 

^ - E ¥ E ^ e [iWO+1) - H) ° W1 /- k+P /2,a / k+ p/2 l0 . (19) 
k a=l,2 



III. SINGLE PAIRING CASE 



First, let us start from explaining our results in pure singlet cases where Wtt = w s t = Wt s — 0. The following 
results remain essentially valid for the corresponding triplet-only cases. In this section, we consider the cases with (i) 
a full-gap Ak = 1, (h) horizontal-line gap nodes Ak = V^cos(2k z ), and (iii) vertical-line gap nodes Ak = v / 2cos(20k)- 
The cases (i) and (iii) correspond to the ordinary s-wave pairing and d x 2 _ y i -pairing ones, respectively, while a pairing 



state 



151 ] proposed for CeRhSi3 [16| and CeIrSi3 [17[ corresponds to the case (ii). 
According to the familiar Hubbard- Stratonovich transformation (l8j . the quadratic term of the GL functional is 
given by 

F w = / d \\ J-|A S | 2 - ]T A :^ a) (n)A s 

L a— 1,2 - 1 

/ dp—- Y (|A k | 2 (cos(pu :E (3o) - iSNsm{pv x Q a))exp(-ipay • II)) 

Wss -V 1 a=±l 



N / d 6 rK 



A s , (20) 



where 



2vrT 



(21) 



sinh(27rTp) ' 

N = Ni + N2, and Qo = 2(j,H/vf is the familiar modulation wavenumber of the vortex-free FFLO state in centrosym 



metric case in low T limit 0, 0). In writing eq.(20), the quantity Qo/yl — J(l — cosfc z ) was simply expressed as Qo 
by assuming a nearly cylindrical FS with a small J (< 1). Validity of this treatment will be explained in relation to 



Fig. 3 and also in sec. IV. The kernel is expressed in terms of <5 a (k, ie) by 



K[ a) {U) = TJ2 I |A k | 2 Sa(k,i£)£ a (-k + n,-i £ ) 

= -f / d Pf(P) E (|Ak| 2 exp(-i^v(n+(-l) a Q i))), 

1 J Pr L1 



(22) 



a=±l 



where II = — iV + 2eA is the gauge-invariant operator for Cooper-pairs, and £ a is the normal quasiparticle energy 
measured from the FS a in zero field. 

Due to the gauge-coupling through II, the pair-field A s (r) will be expanded via basis functions of Landau levels 
(LLs). Before processing the above expression further, the corresponding LL basis function <p n (z,x\0) in the gauge 
A = Hzx will be first determined following the conventional manner of incorporating the anisotropy 7 in the low field 
GL region 19 : The exponential operator in eq.(22) is rewritten in terms of the identity e A+B = e^ A ' B ^ 2 e A e B with a 
constant [A, B] as 



cxp(— \pv ■ II) = cxp(p/ill + — p/!*n_) 

= exp(-|/z|V/2) exp(p^n+) exp(-p/z*II_), 



(23) 
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where 



(7^-17 2 v x ), 



and 



v 2 



(24) 



(25) 



are the creation and annihilation operators of LLs acting on the LL basis functions ip n (z,x\0), where rjj = 1 / \/2eH 
is the averaged vortex spacing originating from the flux quantization. Then, the basis functions take the form 



ip n {z,x\Q) = N Q 



-1)" 



Y,Hn [ ^(z + mkr 2 H ) 



N/2 



"n! 



x exp 



imfca; 



-(z + mkr 2 H ) 2 H — (mkru) 2 cot ( 



(26) 



where H n (w) implies the Hermite polynomial. Under a given superposition of LLs, the structure of a vortex lattice 
is determined by the two parameters k and 9. In the familiar isotropic triangular lattice, we have k^/j = V^^tt/th 
and 9 = tan _1 (\/3/7). More generally, any basis function ip n (z,x\ — zqz) in the gauge A = Hzx satisfies 



,{z - z , x\0) = e izax/r H(p n (z, x\ - z Q z) 



1 



<p„(z,x\ - Zqz) = -=(U + ) n ip (z,x\ - z z). 



(27) 
(28) 



For later convenience, the following formula § for ip n (z,x\0) will be given here 

exp(-| W | 2 /2) / . d 



exp(i/9v • II) (f n (z, x\0) 



p p- 



d{pp) 



x exp(p 2 p 2 /2)(p (z+ y / 2<y- 1 r H pp,x\0). 



(29) 



Using </9„(z, x\0), the "imaginary" term oc sm{pav x Qo) in with a nonzero SN would induce a coupling be- 
tween even and odd LLs. To try to exclude such an even-odd coupling and to make LLs better basis functions 
for diagonalization, the factor exp(— ipuv • II) in eq. (|23p will be written as exjp(ipav x Q) exp(— iperv • TI S (Q)), where 
H S (Q) = II + Qx = — iV + r^ 2 (z + Qr 2 H )x. Correspondingly, the pair-field is expressed as 



A s =^2Y s>n ip n (z + Qr%,x\0). 

n 

Then, using eq. (|29[) . the matrix element appearing in F£ is written as 

J d 2 r[<p nx (z + Qr 2 Hl x\0)]* exp(ipv ■ U s (Q))ip n2 (z + Qr 2 Hl x\0) = exp(-p 2 |^| 2 /2)C nun2 (pfi) 1 



where 



min(ni ,712) 

Cn un3 (w)= 



"o=0 



(ni - n )\(n 2 - n )!n ! 



>(-w*)' 



(30) 



(31) 



(32) 



( s) 

In this way, F 2 V is expressed by 



F. 



00 



N 



= V Y* 



Nw s 



dp 



f(p) 



E 

<7=±1 



p 2 ImI 2 



exp -ipav x 



x |A k | 2 (cos( / 9?; :!; Qo) + i<5A r sin(pcrw a; Qo)) C ni , n2 (pap) 



Y 



(33) 
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As usual, the zero field (mean field) transition temperature T c can be introduced by deleting \/(Nw ss ) through the 
relation 

1 T f 2ttT c , , 

^r ln T c + Jj p ^M^fJy (34) 

To determine H C 2(T) and the pair- field solution giving a free energy minimum at each (H, T) below H C 2(T), we only 
have to diagonalize the above expression of F2 and to determine Y s ^ n giving the lowest eigenvalue under a fixed Q. 

In the conventional GL region in low fields where both Qq and II are small in magnitude, the imaginary term 
disappears if identifying Q with SNQq, and if is diagonalized via the LL basis functions tp n (z + Qr 2 H ,x\Q) in the 
different gauge A = H(z + Qr 2 H )x. According to eq. p7|) . this function is nothingbut exp(— iQx)ip n {z, x\Qr 2 H z). 
Due to the phase factor e~ l( ^ x , this state was often called a helical vortex state [12|. However, <po(z + Qrjj,x\0) 
itself is an Abrikosov triangular lattice in the lowest LL, and the nonzero (J is not practically observable in gauge- 
invariant quantities such as |A s (r)| 2 . As shown in our previous report [20j] , the vortex lattices show a single or 
consecutive first order structural transitions (FOSTs) with increasing field, depending upon the SN value, even when 
assuming Q = SNQq, because the paramagnetic depairing enhanced by increasing the field makes the higher LLs with 
n > 1 active. However, it is unclear whether or not the phase diagrams obtained under the assumption Q ~ SNQq 
are justified in higher fields. Taking account of this point, we have also examined the phase diagram by directly 
optimizing the Q-value at each (if, T). As is seen later in the full gap case, however, a direct optimization of the 
Q-value does not significantly change the resulting phase diagram. For this reason, in obtaining phase diagrams for 
other pairing states, we shall focus later on those following from the relation Q = SNQq. 

So far, we have implicitly assumed that no modulation parallel to H || y occurs in Rashba superconductors, because 
the anisotropic Zeeman term in eq. (fl5|) is not accompanied by k y and hence, does not lead to a paramagnetic depairing 
effect on the spatial variations parallel to H of A s in contrast to the case of the FFLO state in the lowest LL 0|. In 
fact, we have verified this fact concretely in each case of pure singlet pairing. Therefore, throughout this paper, the 
pair-field can be assumed to be independent of y. 

Next, we explain how to evaluate the quartic term of the GL free energy functional. Using the set of Y StTl determined 

(s) ... . . .... . . (s) 

from i<2 ■ an equilibrium vortex lattice structure is obtained by minimizing the quartic term. The quartic term F± 
is, as well as the last term in eq. (|20|) . the sum of contributions from each FS and, in general, takes the form 

(35) 



F 4 (s) = J d 3 r^xi a) ({nj)A;(r 1 )A s (r 2 )A:(r 3 )A s (r 4 ) 



where 

xg a (-k + ir> a) , -ie)g a (k + (n< a) )* - n 2 a \ie), (36) 

where Hj"' = — iV^ + 2eA(rj) + (— l) a Qox. However, when numerous LLs are used in describing A s , it is numerically 
formidable to exactly examine this expression. For Rashba superconductors, however, replacing the original quartic 
term with the conventional GL representation 

if> ~ Pi s) = c s J 'd 3 r|A s (r)| 4 (37) 

with a positive coefficient c s seems to be justified. First, by directly examining F 4 (s) for the cases with only the lowest 
LL and the lowest two LLs, we have found that, in contrast to the centrosymmetric case the overall sign of F^ 
remains positive even in T — > limit. Based on these observations, it is believed that c s > so that the first order 



i? C 2-transition does not occur in noncentrosymmetric superconductors 14j , Further, for the purpose of determining 



a stable structure at each (H, T), even the H and T dependence of c s is unnecessary. On the other hand, even the 
nonlocal corrections arising from the orbital depairing have been neglected in F\ . Explaining this point will be 
postponed until our numerical results are presented in this section. 



To represent |A S | 2 in terms of LLs, it is convenient to use the formula [21| 

< (z, x\Q) Vn2 (z, x\r ) = Awn ( ~ ) F(G, r )<**, (38) 
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where f = z^/jz + xx/y/j, 



F(G,r ) = (-l) mim2 exp 



-Iq^+iG-fo 



(39) 



is the Fourier transform of <fo(z, x\0)tpo(z, x|ro), q = G + ko, ko = y X ro/Vfr, and fo = z^jzq + xxq/^/j. The 
reciprocal lattice vector G is given by 



G = miGi + m 2 G 2 , 
Gi = k^/j(x — 27 cot#), 

G, ' 2 " 



under the condition 



N\G 2 \=M(G 1 -z), 



(40) 



(41) 



where mi, TO2, AT, and M are integers. The condition (j4"T]) ensures a periodicity of an obtained vortex lattice in the 
x-direction. Then, using eq. (j38|) , we have 



|A,(r) 



E Knjs,n 2 [<p ni (Z, X\0)]*<p n2 (Z, x\0) 

E (-l) mim2 ex P (-|r| 2 /2) E ^^^.^(Tje 1 



(42) 



where T = -r H (G x - iG z )/V2. Thus, eq.([53| becomes 



F 



Cs e e 

mi ,772,2 



-|r|- 



and the GL free energy in equilibrium is given by 



2R 



(43) 



(44) 



The equilibrium vortex lattice structure is determined by minimizing F^ or with respect to k and 9. Throughout 
this paper, we present results of the h-t phase diagram obtained in terms of the lowest eight LLs. 

Now, the vortex lattice structures for each pairing state following from the above formulation will be explained. 



Full gap 

First, we explain results in the s-wave pairing case with no gap nodes where Ak = 1. The 5N dependence of the 
phase diagram in the s-wave case has been previously studied based on the assumption Q — SNQq [20|. Here, it will 
be shown that the previous results are not essentially changed by correctly optimizing the Q-value for each field and 
temperature. 

First, let us start from explaining the important role, mentioned in sec. I, of the anisotropy of the Zeeman energy in 
the noncentrosymmetric superconductors. As an example, a phase diagram in the full gap case and in SN — ► limit has 
been shown in Fig. 2(b). The two figures in Fig. 2 have been obtained by assuming J = 0.2 and (J.H^^ (0) / (2ttT c ) = 0.4, 
where is the orbital limiting field at T = in 2D limit. Hereafter, the SN — > limit, in which Q = 0, 

corresponding to the limit of an infinite band width will be often considered to understand an origin of a strange 
vortex lattice structure, and the applied field value will be denoted as h = H/H^F'(0). In Fig. 2(b), intermediate 
lattice structures changing with varying the field appear. They are induced by higher even LLs contributions to A s 
with n = 2m (m > 0), which play enhanced roles in noncentrosymmetric systems through the v x oc k x dependence 
in the Zeeman energy term of eq. (|16[) . Such intermediate phases do not appear in Fig. 2(a) corresponding to the 
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FIG. 3: (Color online) Resulting h-t phase diagrams for (a) SN = 0, (b) SN = -0.003, and (c) SN = -0.2 and (d) -Q/Q 
v.s.h curve taken on H C 2(T) of the figure (c) in the full gap (s-wave pairing) case, where h = H/H^°\o) and t = T/T c . With 
increasing |<WV|, the H C 2-value merely shows a slight increase, while the corresponding change of the vortex lattice structure is 
drastic. The intermediate structures with no reflection symmetry, seen in Fig. 2(b), compete with the novel striped structure, 
(B) of (b), modulating along x perpendicular to both of H and the c-axis and are overcome by the latter at higher SN- values. 
The parameter values nH^\o) / (2ttT c ) — 0.4, J = 0.1, and p c = (207rT c ) — 1 are commonly used. 

familiar centrosymmetic case which is obtained by replacing cos(pv x Qo) in eq. (|33[) with cos(pvfQo) independent of k 
and setting 5N = Q = 0. Thus, the intermediate states in Fig. 2(b) can occur even in Rashba superconductors with 
smaller Maki parameters where the high field state (A) of Larkin-Ovchinnikov (LO) type does not appear. Further, 
the states (B) and (C) in Fig. 2(b) and the continuous crossover between them are also seen in the singlet-triplet (s-t) 
mixed case to be discussed later. 

In Fig. 3, our results in the s-wave case obtained by directly optimizing the Q-value are shown together with the 
resulting Q(h) data obtained along the H C 2(T)-cwve in SN = 0.2 case. All figures in this section including Fig. 3 
have been obtained by using J = 0.1. The H C 2(T)-value is slightly enhanced with increasing \SN\ and thus, \Q\ oc \(\, 
suggesting that an increase of spin-orbit coupling diminishes the paramagnetic depairing. The resulting phase diagram 
in SN = case, Fig. 3(a), is quite similar to Fig. 2(b) (see, however, the next paragraph). By assuming a very small but 
nonvanishing SN, not only the LO type state, (A) in Fig. 3(a), but also the intermediate states (B) and (C) there are 
destabilized by the appearance of the novel modulated state, (B) and (C) in Fig. 3(b), which have reflection symmetry 
in contrast to (B) and (C) in Fig. 3(a) and are stabilized by the hidden Q = Qx-vector or the resulting anisotropy (see 
sec. I). Throughout this paper, we often encounter a compressed square lattice in Rashba superconductors, which is 
created through a field-induced crossover from the stripe- like modulated state, (B) in Fig. 3 (b). As SN is increased 
further, the states (A), (B), and (C) in Fig. 3(a) with no reflection symmetry in the plane perpendicular to H are 
completely lost, and the high field state for SN — 0.2 is the square lattice corresponding to (C) of Fig. 3(b) which 
continuously occurs through a crossover from (B) in Fig. 3(b). Further, even the triangular vortex lattice at higher 
temperatures is compressed with increasing \SN\, possibly reflecting mixings between even and odd LLs induced by a 
nonzero SN, and the only structural transition surviving at higher \SN\ is the FOST between the compressed square 
and triangular lattices. Note that there are two types of continuous crossovers in structure in intermediated fields. 
As well as the crossover expressed as (B) and (C) in Fig. 2(b), this crossover between (B) and (C) in Fig. 3(b) also 
appears in the more general case with s-t mixing of pairing channels. 

Here, the difference in the orientation of the low field triangular lattice between Fig. 3(a) and other figures in Fig. 2 
and 3 will be commented on. First, in Fig. 3, we find that even a small but nonvanishing \SN\ changes the orientation of 
the triangular lattice, and that, at a fixed (and nonvanishing) SN, there is no indication of an orientational transition 
in the triangular lattice region. Below the dashed line in Fig. 3 (a), however, the orientation is found to be the same 
as in Fig. 3(c). This feature is commonly seen in the cases of other pairing states to be given below. On the other 
hand, the difference in the orientation between Fig. 2(b) and Fig. 3(a) can be attributed to the difference in the J-value 
used in calculations, or equivalently, to the strength of the paramagnetic depairing: As the paramagnetic depairing 
is enhanced, higher LL modes with even indices in the pair-field become more effective and change the orientation. 
This statement was justified by separately performing a calculation taking account only of the lowest LL. Besides, 
the absence of such an orientational change in the case with a nonvanishing SN is also consistent with the effective 
reduction of the paramagnetic depairing due to a nonzero SN mentioned in the preceding paragraph. Therefore, as 
far as the uniaxial anisotropy measured by 7 is not so large, the orientation of the low field triangular lattice in real 
systems with nonvanishing \5N\ is expected to keep that of Fig. 3(c). Hereafter, we will not discuss this possibility of 
an orientational transition any longer. 

Quantitatively, there are some differences between the present Fig. 3 following from the Q-optimization and Fig. 3 
in Ref.(20j where Q — SNQq was assumed. For instance, in the latter for SN = 0.003, the LO-like state, (A) in 
the present Fig. 3(a) and (b), survives over a broader field range compared with that in the corresponding one in the 
former. However, except such quantitatively subtle differences, there were no notable differences due to the Q-value 
in the phase diagrams and their SN dependences. This evidently shows that the analysis in Ref.jl^] trying to take 
care of the approximation on the Q- value by including the eight LLs is justified. Further, judging from this fact that 
the vortex structure is not sensitive to the Q-value, our neglect of the fc z -dependence accompanying Qq in eq.(20) is 
also believed to be safely valid. 

(s) 

Here, based on the figures in Fig. 3, our replacement of the original quartic term F± with the conventional local 
expression _F 4 will be discussed. This replacement is safely valid in centrosymmetric superconductors with a large 
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paramagnetic effect 22J, |23[. Even in the present Rashba case, the same thing should hold. For smaller \SN\, the 
paramagnetic effect is stronger, and intriguing modulated vortex structures tend to appear. Although there might 
be possibility that the validity of this local approximation is subtle in intermediate fields, the structural changes in 
the case with nonzero SN are smooth so that most of FOSTs in SN — case are changed into crossovers. In such a 

(s) ~ (s) 

crossover regime, a large deviation between the results of F± ' and of F± ' is not expected. For this reason, we believe 

(s) 

that the use of the local approximation for F£ is qualitatively valid and does not lead to a significant error in our 
results on the vortex lattice structure. 



Horizontal line nodes 



Here, phase diagrams in the case of a superconducting gap with horizontal line nodes will be briefly ex plai ned. 
Recently, such a nodal gap has been proposed as a model of CeRhSi3 and CeIrSi3 [15(, and, following Ref.[15[, we 
choose the gap function Ak = \/2cos(2fc z ). Figure 4 includes the resulting phase diagrams in this case. Clearly, the 
obtained phase diagrams are essentially the same as those in the full gap case. This is due to the fact that, in the 
angular average over each FS, the k z dependence of the gap function does not directly couple to the k x dependence 
in the Zeeman energy. Thus, the same thing should hold for any gap function dependent only on k z . Therefore, it 
appears that, when the s-t mixing is negligible, the presence of a horizontal line node in the superconducting gap 
cannot be judged from the resulting vortex lattice structure. 



Vertical line nodes 



In contrast to the preceding case, the momentum dependence in the gap function directly couples to that of the 
Zeeman term when the gap nodes consist of vertical lines, leading to a drastic change of vortex lattice structure. This 
will be explained here in the d x 2 _ y i -pairing case where Ak = %/2cos(2</>k). 

In this d x 2 _ y 2 -pairing case, situation changes depending on whether the applied field H is along the nodal direction 
or antinodal one. Differences between these two cases already appear in the -ff^-curves shown in Fig. 5. In H parallel 
to an antinode, the H C 2(T) value is slightly enhanced compared with that in the full gap case obtained in terms of 
the same set of parameter values. In contrast, H C 2(T) in H parallel to a gap node is remarkably depressed compared 
with the full gap curve. This strong anisotropy in ff c2 is a consequence of the coupling in the momentum dependence 
between Ak and the anisotropic Zeeman term. 

A more remarkable difference is seen in the resulting vortex lattice structures in the two field configurations. In H 
parallel to an antinode, the vortex lattice structure is qualitatively the same as in the preceding two cases, while, in 
H parallel to a gap node, there are no intermediate states in SN = limit, and the high field state is of the LO-type 
with an unidirectional modulation parallel to z, i.e., the c-axis. In this case, the direction of the modulation in the 
LO-like state is pinned by the vertical line nodes. The resulting vortex lattices always have reflection symmetry in 
the plane perpendicular to H in contrast to the intermediate phases in Fig. 3(a) and Fig. 4(a). This fact may have 
crucial consequences in the cases with a significant mixing of a d-wave component with vertical line nodes and the 
corresponding /-wave one (see sec.V). 

To understand whether this vertical LO state survives for realistic SN values, we have also examined the corre- 
sponding phase diagrams for nonzero SN values and have found that, for \SN\ > 0.1, the vertical LO state is absent 
even close to H c2 (0). As Fig. 6 shows, however, the novel striped state with modulation perpendicular to the c-axis, 
corresponding to (B) in Fig. 3(b), occupies a much narrower region compared with that in Fig. 3(b) in the full gap 
case, because the LO-like state competing with this novel intermediate state is supported in this case by the vertical 
line nodes. Thus, we expect that the region in which the novel modulated state replaces the vertical LO-like state is 
narrower even for more realistic SN values, and thus that the LO-like state may be observable as a high field state 
in this case in contrast to that in the preceding two cases. For these reasons, we believe that the presence of vertical 
line gap-nodes can be anticipated by investigating vortex lattice structures. 



FIG. 4: (Color online) The resulting h-t phase diagrams for (a) 5N — and (b) SN = —0.1 in the case with Ak = \/2cos(2fc z ). 
Other parameter values are the same as in Fig. 3. 
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IV SINGLET-TRIPLET MIXED CASE 



Quasi 2D case 



In the preceding section, the singlet (s)-triplet (t) mixing, which is usually present in noncentrosymetric supercon- 
ductors with nonzero SN, has been neglected by assuming one of the two channels to be dominantly attractive. This 
approximation is valid for a vanishingly small \5w\, i.e., as far as one of w ss and wtt is small enough (see eq.Q). 
However, when the ratio Wtt/w S s is of order unity, this s-t mixing drastically changes the H-T phase diagram even if 
\SN\ is vanishingly small. That is, the s-t mixing is measured by a finite wtt/w ss , he., eq.(|4|). in nonzero fields rather 
than a nonzero 5N [ll[ in zero field case, as a result of the fact that the pair-field in nonzero fields is intrinsically 
spatially varying. 

In this section, roles of the s-i-mixing will be first examined in details for quasi two-dimensional (Q2D) systems 
with a large 7. For simplicity, we focus on the case of a mixing of s-wave and p-wave pairings. In this Q2D case, 
the momentum dependence of |gk| appearing in some places may be neglected to simplify our evaluation of the free 
energy. To clarify the details of this treatment, let us first start from introducing the pair-field on each of the split 
FSs in the s-t mixed case. As the expressions of pair-field operators f p ' (j = s and t) suggest, the (spatially varying) 
energy gap on the FS a is generally given by 



A s -(-ir| gk |A t 



(a = 1,2) 



(45) 



which is accompanied by the momentum dependence gk of the spin-orbit coupling even after having been separated 
from Ak-dependence, where A s (A t ) is the singlet (triplet) gap corresponding to In Q2D case, the factor 

|gk| in A a is replaced by unity, and 



A „ A (o) = A.-(-l)°A t 



(46) 



will be used in this section. 



Derivation of the quadratic GL term i^Q 20 ) [ s almost the same as in the preceding section once Aj (j — s and t) 



are expressed via Ai through eq. (l46|) . and we obtain 



F 2 (Q2D) = / d 3 r 



E 



| A (0)|2_ 2(A (0) r ^) (n)A (0) 



(to 1 ) ss - (to 1 ) t 



[A*A 2 + (c.c. 



(47) 



According to the expression (|22|) of it is natural in this case to choose the gauge in the manner depending on 

each FS and to represent the pair-field in the form 



Ai 0) =5> a ,^„(r a |0), 

n>0 

with r a = r + (— IY'Qqt^z. Note that the gauge-invariant gradient corresponding to ip n (r a \0) is 

II a = -iV + r H 2 {z + {-l) a Q r 2 H )x. 
Using the formula (|2"5|) , the term including the p-integral in eq.l|47p becomes 

J d 3 r(Ai ))*^ a) (n)A(°) =N a V Y, Y* ni Y a , n2 dpf(p)(e-iM 2 ? 2 Re£ nun2 ( W ) 



(48) 



(49) 



(50) 



FIG. 5: (Color online) The resulting h-t phase diagrams in the d :c 2_ !/ 2-pairing case in magnetic fields applied along (a) a node 
and (b) an antinode of the energy gap. For both figures, 5N = was assumed, and other parameter values are the same as 
those in Fig. 3. Note the modulation parallel to the c-axis and the absence of intermediate phases in (a). 
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FIG. 6: (Color online) The resulting h-t phase diagram in the d ;c 2_ H 2-pairing system with 5N = —0.003 in magnetic fields 
applied along a node. The same parameters are used as in Fig. 5 

To calculate the off-diagonal (last) term in eq. (j4"Tj) , we will directly use the formula ([55]) . and consequently, 

|d 3 r(Al 0) )*Af = Y,K ni Y2,n 2 J dzd Xip : i (v 1 \0) i p n2 (r 1 \2Q r%z)e- i2 ^ (51) 



V E Y* ni Y 2 , n2 e-^o r "£ n2 , ni (^Q r H ), 



where the property (127p was used. 
Therefore, eq. (|47|) takes the form 



^2 



(Q2D) 



V 

-2N, 



= E KAn, \( {W ~ 1)S ° + 2 {W ' 1)tt ~ (-D>-%) S nun2 
a,ni,Ti2 

roo - 

a / dpf(p)U-^ 2p2 ReC ni ,„ 2 (M) 

■J Pa 

-N(Sw)- 1 ]T (Yl ni Y 2 . n2 e-^ 2r * C n2 , ni { ^Qr H ) + (c.c.)) . (52) 

Here, it is important to note that the paramagnetic effect appears through the FFLO wavenumber Qq only in the 
last term, i.e., the cross term between different FSs. Thus, in \5w\ — > oo limit, i.e., when w ss ~ Wtt (see eq.@), the 
paramagnetic effect is lost irrespective of the u^-value, and the orbitally-limited situation is realized. The parameter 
\5w\ measures the magnitude of the s-t mixing in nonzero fields. In the present Q2D model, the energy gap on the FS 
with a smaller density of states vanishes (see, for instance, Fig. 8 shown below) in \Sw\ — > oo limit. The single-pairing 
case, that is, the pure singlet or pure triplet case corresponds to the case with vanishing 6w, while the single and 
triplet channels equally contribute and are competitive with each other when \6w\ — > oo. 
The zero field transition temperature T c is given by Wij through the expression 



(W ~ % 2 ; ( ^ % = [dpf{ P )\T^ + yW 2 + (sn J~dpM\^. + ^) 2 - (53) 

It is not difficult to verify that the above expression is equivalent to eqs.(28) and (31) in the impurity-free case in 



Ref.[24| 



Regarding the quartic term, the method in the single pairing case will be directly used. The quartic term F^ 2D ^ 
is given by ea. (|35p with A s replaced by , and, for a similar reason to that in the single pairing case, it will be 
replaced by its local expression 



C2 |d 3 r5> Q |A(°)(r)| 4 , (54) 



where c 2 is positive. Then, the quartic term to be used for determining the lattice structure is given by 



P (Q2D) 

V- = C2 ^> E e~' r 

a m i , m 2 



(55) 



and the GL free energy in equilibrium is given by 



(p(R)) 2 

r (R) = -^iwr- (56) 



2F (R) 



We show, in Figs. 7 and 8, examples of the resulting phase diagrams and vortex lattice structures in the case with 
a mixing of s-wave and p-wave components and with a Q2D (cylindrical) FS. In those figures, the vortex lattice 
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FIG. 7: (Color online) The resulting h-t phase diagram in the s-wave and p-wave mixed case based on the Q2D approximation 
(|46|) . The used parameters are J = 0.2, SN = and Sw = 1.25. For each point in the phase diagram, the four images in real 
space of | Ai|, IA2I, |A S |, and |A t | are shown from top to bottom. 

structures at several selected points in each h-t phase diagram are represented as real space patterns of |A^| on FS1, 
\A^\ on FS2, |A S |, and |A t |. In general, when \SN\ is essentially zero so that both of the two FSs contribute equally 
to superconductivity, as shown in Fig. 7, the vortex positions in A J ' do not coincide with those in A 2 , reflecting the 
fact that the spatial pattern of |A S | tends to become quite opposite to that of |Aj|. In this case, when A s consists of 
even LLs, At is expressed only by the odd LLs, and vice versa. On the other hand, with increasing \SN\, one of the two 
FSs dominantly contributes to superconductivity, and, according to eq. ([^|) . the (nearly) zero points of |A S | coincide 
with those of |A t | (see Fig. 8). When one of the two FSs primarily determines superconductivity, the paramagnetic 
depairing is significantly reduced, because the paramagnetic effect on a single FS is trivially gauged away [14j (see 
also Fig. 1(b)). As in the case with a single pairing component, an increase of \SN\ results in a significant mixing 
between the even and odd LLs, and, as in Fig. 8, |A S | tends to show similar spatial patterns to |A t |. 

Figure 7 corresponds to a phase diagram of the case with a slight inclusion of a p-wave pairing component in the 
pure s-wave case. Here, SN is set to be zero, and hence, this figure is comparable with Fig. 3 (a) and (b). Similarly 
to the difference between Fig. 3(a) and Fig. 3(b) induced by a slight increase of \SN\, the intermediate state (C) in 
Fig. 7 with no reflection symmetry are limited to a narrow region and dominated by the stripe- like modulated lattice 
(A) and (B). However, the structure at (B) in Fig. 7 is not a square lattice appearing as a crossover from the striped 
structure (A) but rather a triangular lattice which can be obtained by rotating another triangular lattice appearing 
in lower fields just below FOST4. The triangular lattice in intermediate fields is also present even for larger \8w\, 
i.e., even when the s and p-wave components are more significantly mixed, while it is lost as |<5iV| is larger, as can 
be seen in Fig. 8 where a more s-t mixing and a larger \SN\ than in Fig. 7 were assumed. A typical SN dependence 
°f H c2 (T)-curve including that of Fig. 8 has been given in Ref.[3|. In the SN = limit, the h- value corresponding 
to H c2 (0) is close to 3.0. Such a much larger H c2 ({)) than that of Fig. 3(a) is due to the s-t mixing, which clearly 
plays more dominant roles than a nonzero SN for enhancing H C 2- Nevertheless, Fig. 8 has similar features to those 
of Fig. 3(c). For instance, in both Fig. 3(c) and Fig. 8, the triangular lattice near T c shows an anisotropic structure 
compressed along the c-axis, while the vortex structure in higher fields is an anisotropic square lattice created from 
the novel striped lattice (B) in Fig. 3(b). Therefore, in the present case with a cylindrical FS with a negligibly small 
corrugation, an increase of a s-t mixing plays qualitatively similar roles to an increase of the magnitude of the spin- 
orbit coupling \SN\ in the h-t phase diagram, and, in a realistic situation where both Sw and \SN\ are nonvanishing, 
vortex lattice structures with no reflection symmetry such as (B) in Fig. 3(a) are expected not to occur. 

In (C) of Fig. 8, the image of |A S | is much brighter than that of |A t |, while both of them in (A) are almost the same 
as each other. The former feature in lower fields is a reflection of the fact that, in zero field, the singlet component 
is the dominant pairing state, and a small \SN\ induces the triplet componet, while, with increasing field, the role of 
inducing a s-t mixing is played not by \SN\ but rather by the nonvanishing Sw. In particular, at high enough fields 
and in low temperature limit, the vortex structure is an anisotropic square lattice oriented along the c axis for any 
SN, implying that the phase diagram there is sensitive not to SN but to \5w\. In the next subsection, however, these 
conclusions in Q2D case are found to be changed for more three dimension(3D)-like FSs. For instance, the h-t phase 
diagram for a more 3D-likc FS seems to have a much stronger SN dependence than that seen above. 

In obtaining Fig. 8, we have assumed SN < and w s t > 0. Since w s t generally depends on the higher energy cutoff, 
the results following from diagonalization of F^® 20 ^ are quantitatively affected by the details of w st . In fact, if w st is 
zero or negative, |Ai| rather than |A 2 | should be larger. The opposite result to this, seen in Fig. 8, is a consequence of 
a positive w st . However, we have reexamined Fig. 8 by changing the sign of w st and have found that the field induced 
changes of the vortex lattice structure remain qualitatively the same as those in Fig. 8. Based on this fact, we shall 
assume hereafter the vortex lattice structure to be qualitatively insensitive to the sign of w s t ■ 



FIG. 8: (Color online) The resulting h-t phase diagram in the s-wave and p-wave mixed case based on the Q2D approximation 
(|46|) . The used parameters are J = 0.2, SN = —0.1 and Sw = 5. 
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More 3D-like case 



In this subsection, the Q2D approximation in the last section is relaxed, and effects of the corrugation parallel to 
the c-axis of the Q2D Fermi surface will be incorporated. Then, the /cz-dependence in A a neglected in the last section 
needs to be included. In contrast to the weak k z dependence accompanying the parameter Qq, neglected for simplicity 
in eq.(20), this k z dependence may lead to a change of the degree of mixing of even and odd LLs which affects the 
vortex lattice structure. To simplify our treatirierrt, the factor |gk| in A a will be approximated by 



|gk| = V 1 - J ( 1 _ cosk z ) 
Jcosfc, 



I — J 



(57) 
(58) 



2 \/l — J 



Then, using 



A s -(-1)%/1- J A 



(59) 



A a is expressed as 



A a = a(°) - (~ir Jcoskz A t 



2^/2(1- J) 

(i + 0Af)-aS 0) , 



where a and b = 1 or 2, a ^ 6, and 



Jcosfc, 



? = 



(60) 



(61) 



4(1- J) 

First, let us start from rewriting terms dependent on Wij in the GL quadratic terms (see eq. (|47p ) into the form 
2 E ((™ -1 )«A* A j) = [(^ 1 ) ss + (^ 1 ) tt + 2(^- 1 ) st ]|A(° ) | 2 + [( w - 1 ) ss + (*- 1 ) tt -2(^ 1 ) st ]|A(° ) | 2 

~2[(w- 1 ) ss - (w- 1 ) tt ]Re(A(° ) *Af) (62) 



i,j=s,t 



expressed by A a , where (w 1 )u — (w 1 )tt/(l — J), and (w 1 ) s t — (w 1 ) s t/ V 1 — J- The remaining term including 
the kernel K^ 1 is given by 

-2^(A:^ a) (n)A a ) = -2 / dpf(p)Y / N a {A:e- s ^ vn -A a 

a J Se ,a 

d P f(p)Y, N * 

s e ,a 

x^(l + ? ) 2 (A^°^)*e _SEipv ' na A' ^ + ^e^'^Qo^A^^e^'^'^A^ 
-?(! + ?) ((A^)*e- SEipv - n »A[ 0) + (A( 0) )*e" Seipv - n "Ai°))y 



(63) 



Note that this term is diagonal with respect to A a , while it is not diagonalized any longer in the A^-representation. 
Then, by representing Ai°\ as in eq. (|48[) . in the form 



AW = ^F a ,„^„(r a |0), 



(64) 



n>0 



15 



the cross term between A^ and A^ in eq. (|63|) becomes 



J dM^ye-^^AM = £ Y h * n J a , n2 J nun2 ((-l)V^Qor-H, 



SsPP , 



(65) 



«1 ,12 



where 



■ki,n a ((-l)V27<3orH,s e Mp) = / d 3 r<p* ni fa, |0)e— Vtw (r 10) 



1 1 I |2 2 



= e 2 1 



-s £ /z p + s £ 



d(fMp) 



\Jnx\n 



d{fip) 



r H + s E fip I e 



(66) 

The last equality in eq. (|66j) can be proved inductively. In this way, the quadratic GL free energy in the present case 
is expressed in the form 



— - V Y* Y 

2Y ~ Z_^i I a,n 1 I a,n 2 



(-lHuT 1 )* S ni , n 



1 dpf{p)(e--M 2 p 2 { Na (i + ,) 2 Re£ ni ,„ 2 ( MP ) + AT 6? 2 Re (e^ 1 ^ 1 ^*^,^))}) 



where 



(JVi +iV 2 ) (^-^-^^^^(^Qorff) 
dpf(p)^(l + ,){N 1 (j: 2tni (^Qor H ,pp) + j;^ ni (^Q r H ,-pp) 
+N 2 (^J nurl2 (- ^/2jQ r H , p,p) + J«i,n 2 (- V^Qorf/j-wO)}) + (c-c.) 

2(JV! + iV 2 ) 



5iu 



^ss - (w 1 ) t 



(67) 



(68) 



is the measure of the s-i mixing redefined within the treatment in this section. Here, since Wtt is not the bare attractive 
interaction potential, the full mixing of the two pairing channels does not coincide with the limit in which Sw diverges. 
The above F 2 -expression implies that the zero field transition temperature T c is determined from 



2{Nx + N 2 ) 



= (1 + m 2 ) / dpf(p)\ T=Tc ) + 



™ 2 d P f{p)\T=T c -{8w)- 1 



jurist 
N x +N 2 



+ 6N dpf{p)\ T=Tc 



where 



— 



(69) 



(70) 



4(1- J) 

In contrast to F^ 2D \ the paramagnetic effect in the present free energy does not disappear even when 5w diverges. 

Since the parameter Qq appears even in the diagonal terms with respect to Al , it is not easy to, in advance, prescribe 
the situation in which the orbital-limiting is realized. 

Again, the local approximation will be used for the quartic term of the corresponding GL free energy functional to 
determine stable vortex structures. The quartic GL term to be examined is 



fW* |d 3 r^7V a (|A a | 4 ), 



(71) 
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FIG. 9: (Color online) The resulting h-t phase diagrams in the s-wave and p-wave mixed case for (a) 5N = and (b) SN — —0.1 
obtained in terms of eq. (|60[l . The dashed portion of FOST2 is not identified due to a numerical difficulty. The used parameters 
are J = 0.35 and Sw = 5. 



where 



|A a | 4 = (1 + .) 4 |A(°)| 4 + , 4 |Ai°Y - 4,(1 + 3 |A<°>| 2 Re(A<°>'A<°>) - 4, 3 (1 + 0|A<°>| 2 Re(A<°>'A<°>) 
+2, 2 (1 + ,) 2 [|Ai°)| 2 |A^| 2 +Re((Ai ^Ai°)) 2 ) +|A(°)*Ai°)| 2 ; . 

To examine eq. (|72[) in more details, we first rewrite Ai A^, (a, a' — 1, 2) in the form 
E ^„ 1 ^,„ 2 |d 2 r< 1 (r a |0)^ 2 (r a ,|0)e- iG - f 



m ,712 
(-1)" 11 ™ 2 exp 



\r nn ,\ 2 
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(73) 



where eq. ([38| was used, and 
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Using eas.([72]l and (73]), eq.j7T|) becomes 
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Substituting Y a n determined from F 2 into eci. (f?3")) . the stable lattice structure can be determined from p( R) . Further, 
just like in Figs. 7 and 8 in Q2D case, we will present not only the resulting phase diagram but also the spatial 
variations of |A a |, |A S |, and |A t | at some selected points in an h-t phase diagram. According to the expressions given 
so far, they are given by 



|Aa| 2 = 

IAJ 2 = 
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(77) 
(78) 



A typical example of <5A-dependences of the h-t phase diagram at the same Sr- value is shown in Fig. 9. The value 
Sw — 5 is estimated by assuming u> c /T c = 10 to roughly correspond to Sw — —2, where lo c is the higher energy cutoff 
for the pairing. For this reason, the -ff C 2-value suggested in Fig. 9(a) is comparable with those in Fig. 2 and is much 
smaller than the corresponding one in Q2D case with Sw = 5. In spite of this, the SN dependence in Fig. 9 with a 
more 3D-like FS is dramatic compared with that in Q2D case: Even a small \SN\ leads to an -ff c2 (T) curve close to the 



FIG. 10: (Color online) Extended views of |A 2 (z,a;)| on FS2 at (A) (top) and (C) (bottom) in Fig.9(b). 
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FIG. 11: (Color online) The resulting h-t phase diagram in the s-wave and p-wave mixed case obtained in terms of eg . (I60fl . 
In the intermediate region between FOST2 and FOST6, the vortex lattice has a highly compressed triangular structure. The 
used parameters are J = 0.35, 6N = 0, and 5w = 0.454. 

orbital limit, although we have checked that the H C 2{T) curve in Fig. 9(b) lies slightly below that in the orbital-limited 
case. 

More important differences from those in Q2D case are seen in the resulting vortex lattice structures. According 
to the results in Q2D case, a slight inclusion of a finite Sw has similar roles to those of a finite SN, and the scenario 
suggested by Fig. 3 was that the states of LO type with no reflection symmetry are destabilized. However, the 
corrugation of the cylindrical FS, or a 3D-like FS seems to destabilize rather the novel striped modulation induced 
by the finite SN and appeared as (B) and (C) in Fig. 3(b). In fact, it seems based on some phase diagrams we have 
numerically obtained that the structures (B) and (C) in Fig. 3(b) with reflection symmetry are close in energy to 
another structures in intermediate fields, (B) and (C) in Fig. 3(a) with no reflection symmetry and thus that the 
crossover between the former structures competes with the corresponding one between the latter structures. The 
former is supported in part by the nearly straight cylindrical FS, while the corrugation of the cylindrical FS or a 
3D-like FS favors the latter. We stress that such a competition is absent in the single pairing case in sec. Ill because 
it is the momentum dependence in eq. (|45[) which induces such a competition between two kinds of modulated states. 

In fact, Fig. 9(a) should be compared with Fig. 7: For instance, the square lattice near H C 2(0) in Fig. 7 is replaced 
in Fig. 9(a) by that of LO type, and the roles of the two kinds of structures (one with reflection symmetry and the 
other of LO type with no reflection symmetry) in intermediate fields in Fig. 7 are precisely exchanged in Fig. 9(a). 
Further, Fig. 9(b) shows that, with increasing \SN\, the region of the triangular lattice in intermediate fields shrinks in 
contrast to the strongly anisotropic triangular lattice near T c , and that the high field region above FOST1 in Fig. 9(a) 
disappears. Consequently, far from T c , the only stable structure in Fig. 9(b) is a strongy anisotropic and tilted vortex 
lattice with no reflection symmetry. As Fig. 10 show, the vortices in Fig. 9(b) have cores compressed along the c-axis. 
With decreasing the field, the tilt angle of the stripes in Fig. 9(b), which is a vestige of the FFLO modulation in (A) 
of Fig. 9(a), decreases. Since the lattice structure there is close to the square symmetry rather than the hexagonal 
one, however, an FOST, just like FOST2 in Fig. 3(a), inevitably occurs to transform into the triangular lattice (C). 
This example also indicates a similar role of Sw to that of SN. We also note that phase diagrams similar to Fig. 9(b) 
have been obtained quite often in our numerical calculations. For instance, even for Sw — 0.4 which seems to be a 
value closer to the orbital limiting, we have obtained the results similar to Fig. 9(b) irrespective of the used SN-value. 

We have not examined a phase diagram for quite a small \SN\ interpolating Fig. 9 (a) and (b). Based on the 
above-mentioned similar roles of Sw and SN, however, it is valuable to examine the SN = case with a larger s-t 
mixing than that in Fig. 9(a). For this reason, we show such an example in Fig. 11. It seems, except the presence of 
FOST4 and FOST5 there, that Fig. 11 interpolates between Fig. 9(a) and (b). In fact, these two FOSTs are expected 
to change into crossovers once SN becomes nonzero, because the roles of even and odd LLs are exchanged through 
FOST4 and FOST5, and an even-odd LL mixing due to a nonzero SN, as in Fig. 3, tends to change an FOST into a 
crossover. Therefore, it is natural to expect that, in the situations interpolating Fig. 9(a) and (b), the intermediate 
triangular lattices are simply lost with increasing \SN\. 

V SUMMARY AND DISCUSSION 

In this paper, possible vortex lattice structures in noncentrosymmetric Rashba superconductors have been studied, 
and, as a result of the anisotropic Zeeman effect peculiar to Rashba superconductors, the vortex structure was found 
to change depending on the pairing symmetry. Through our calculations for several model pairing states, three types 
of sequences of field-induced structural crossovers have been found to appear in superconductors with the ASOC of 
pure Rashba type in intermediate fields depending on the value of a normalized ASOC and on the pairing state: 1) 
LO-like structure with no reflection symmetry and a field-induced rotation of its orientation, 2) novel striped lattice 
modulating along the Qo-direction and its crossover to a compressed square lattice, and 3) intermediate triangular 
lattices differing from the familiar one near T c and in lower fields. However, the LO structure occurring in the d x 2_ y 2- 
pairing case under a field parallel to the gap nodes is exceptional and has a modulation parallel to the c-axis and 
reflection symmetry. 

To obtain close correlations between the vortex structure and pairing symmetry, a detailed analysis taking account 
of a more realistic band structure will be neccesary. Nevertheless, based on the numerical study we have performed 
so far, the following two conclusions are expected to be unaffected by refining the starting microscopic model. Below, 
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we focus on realistic cases with nonvanishing \SN\ ~ 0.1. 

In the case where the s-t mixing is negligible, a modulated state with weak stripes parallel to the c-axis or an 
anisotropic square vortex lattice is realized in the intermediate and high field ranges, depending on the situation, and 
is expected to have reflection symmetry. In the presence of vertical line gap-nodes parallel to the c-axis, the resulting 
high field state is affected by the gap nodes and may be the LO-like vortex lattice with stripes perpendicular to the 
c-axis if the field is parallel to a gap node. 

In the case with a significant amount of s-t mixing, the paramagnetic depairing effect is reduced irrespective of 
the pairing state, leading to an enhancement of H C 2, while the vortex lattice structure seems to depend on the 
dimensionality of the Fermi surface: For the 2D-like case in which FS takes the form of a nearly straight cylinder, the 
resulting vortex lattices seem to always have reflection symmetry and to yield the structural crossover 2) indicated 
above. In the case with a more realistic FS such as a corrugated cylinder, however, the resulting vortex lattice in 
higher fields has no reflection symmetry reflecting the crossover 1) suggesting the presence of the LO-like state in 
SN — > limit. These scenarios are not satisfied in superconductors with the vertical line gap- nodes (|| c-axis) and 
under a field parallel to a gap-node, where the tilted structures with no reflection symmetry do not appear, reflecting 
a pinning of the striped structure via the gap nodes. 

Finally, comments relevant to real experiments are in order: 

Experimentally, an imaging of a vortex lattice can be seen, for instance, through neutron scattering measurements. 
In such an experiment, however, the structure is detected as a flux density distribution which, in turn, is determined 
by a spatial distribution of the supercurrent. Although we have not calculated the supercurrent density in the present 
work, the resulting flux density is, roughly speaking, proportional to the summation J2 a =i 2 \^a\ 2 so that the spatial 
patterns shown in the figures in the preceding sections are essentially detectable. 



Among the existing noncentrosymmetric superconductors, CeRhSi3 [161 ] and CeIrSi3 [171 ] seem to correspond to the 
case with a negligibly small s-t mixing because they show significantly reduced H C 2(T) in H _L c compared with that 
in H || c. If, as in the pairing model proposed recently [151], the pairing state has no vertical line gap-nodes, the 
present results imply that the resulting vortex lattices should keep reflection symmetry, and that no unusual vortex 
dynamics is expected to appear in these materials (see below). 

By contrast, in CePtsSi, nearly isotropic -ff C 2-curves [25| have been previously determined experimentally which 
suggest that the paramagnetic depairing is weak in this material. Based on the present results, this implies that 
its pairing state has a significant s-t mixing, or that the bare paramagnetic effect is negligibly weak. If the latter 
possibility is correct, a natural guess is that the vortex state will be a conventional orbital-limited one, which cannot 
explain the recent interesting observation of an extremely small magnetic decay rate in CePtaSi in H |j a [2r3 ] without 
extrinsically assuming the presence of twin boundaries. On the contrary, if the former case is valid, and the pairing 
state has no vertical line gap-nodes, the resulting vortex lattice has no reflection symmetry (see Fig. 9(b)). This 
suggests the presence of two domains of vortex lattices in real CePtsSi [27j ]. although the pairing state of this material 
is expected to have time reversal symmetry, and it is possible that the observation in Ref . [261 ] is intrinsically explained 
without invoking [28| extrinsic twin boundaries . To clarify this point, similar magnetic measurements in H | c and 
in other Rashba superconductors such as CeRhSi3 and CeIrSi3 are to be performed. 
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